Simple and Effective Dimensionality Reduction for Word Embeddings

Word embeddings have become the basic building blocks for several natural language processing and information retrieval tasks. Recently, there has been an emphasis on further improving the pre-trained word vectors through post-processing algorithms. One such area of improvement is the dimensionality reduction of word embeddings. Reducing the size of word embeddings through dimensionality reduction can improve their utility in memory constrained devices. In this work, we devise an algorithm that effectively combines PCA based dimensionality reduction with a post-processing algorithm, to construct word embeddings of lower dimensions. Empirical evalutions on 12 standard word similarity benchmarks show that our algorithm reduces the embedding dimensionality by 50%, while achieving similar or (more often) better performance.